from sklearn_benchmarks.reporting.hp_match import HpMatchReporting
import pandas as pd
pd.set_option('display.max_colwidth', None)
pd.set_option('display.max_columns', None)
pd.set_option('display.max_rows', None)

scikit-learn vs. ONNX Runtime

reporting = HpMatchReporting(other_library="onnx", config="config.yml", log_scale=True)
reporting.make_report()

We assume here there is a perfect match between the hyperparameters of both librairies. For a given set of parameters and a given dataset, we compute the speedup time scikit-learn / time onnx. For instance, a speedup of 2 means that onnx is twice as fast as scikit-learn for a given set of parameters and a given dataset.

KNeighborsClassifier_brute_force

onnx (1.10.1) vs. scikit-learn (1.0.dev0)

Speedup barplots

All estimators share the following parameters: algorithm=brute.

Raw results

predict

	estimator	library	function	n_samples_train	n_samples	n_features	mean_duration_sklearn	std_duration_sklearn	iteration_throughput	latency	n_jobs	n_neighbors	accuracy_score_sklearn	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
0	KNeighborsClassifier_brute_force	sklearn	predict	100000	1000	100	1.944	0.052	0.000	0.002	-1	1	0.676	16.933	0.006	0.676	0.115	0.115	See	See
1	KNeighborsClassifier_brute_force	sklearn	predict	100000	1	100	0.023	0.003	0.000	0.023	-1	1	0.000	0.303	0.005	0.000	0.077	0.077	See	See
2	KNeighborsClassifier_brute_force	sklearn	predict	100000	1000	100	2.855	0.036	0.000	0.003	-1	5	0.743	17.065	0.024	0.743	0.167	0.167	See	See
3	KNeighborsClassifier_brute_force	sklearn	predict	100000	1	100	0.024	0.003	0.000	0.024	-1	5	1.000	0.303	0.006	1.000	0.078	0.078	See	See
4	KNeighborsClassifier_brute_force	sklearn	predict	100000	1000	100	2.094	0.002	0.000	0.002	1	100	0.846	16.869	0.026	0.846	0.124	0.124	See	See
5	KNeighborsClassifier_brute_force	sklearn	predict	100000	1	100	0.019	0.000	0.000	0.019	1	100	1.000	0.308	0.005	1.000	0.062	0.062	See	See
6	KNeighborsClassifier_brute_force	sklearn	predict	100000	1000	100	2.910	0.037	0.000	0.003	-1	100	0.846	17.044	0.025	0.846	0.171	0.171	See	See
7	KNeighborsClassifier_brute_force	sklearn	predict	100000	1	100	0.024	0.002	0.000	0.024	-1	100	1.000	0.308	0.007	1.000	0.078	0.078	See	See
8	KNeighborsClassifier_brute_force	sklearn	predict	100000	1000	100	2.098	0.011	0.000	0.002	1	5	0.743	17.201	0.023	0.743	0.122	0.122	See	See
9	KNeighborsClassifier_brute_force	sklearn	predict	100000	1	100	0.019	0.000	0.000	0.019	1	5	1.000	0.305	0.009	1.000	0.063	0.063	See	See
10	KNeighborsClassifier_brute_force	sklearn	predict	100000	1000	100	1.173	0.005	0.001	0.001	1	1	0.676	17.242	0.010	0.676	0.068	0.068	See	See
11	KNeighborsClassifier_brute_force	sklearn	predict	100000	1	100	0.018	0.000	0.000	0.018	1	1	0.000	0.304	0.008	0.000	0.060	0.060	See	See
12	KNeighborsClassifier_brute_force	sklearn	predict	100000	1000	2	1.789	0.027	0.000	0.002	-1	1	0.845	3.752	0.008	0.845	0.477	0.477	See	See
13	KNeighborsClassifier_brute_force	sklearn	predict	100000	1	2	0.007	0.003	0.000	0.007	-1	1	1.000	0.239	0.004	1.000	0.028	0.028	See	See
14	KNeighborsClassifier_brute_force	sklearn	predict	100000	1000	2	2.783	0.015	0.000	0.003	-1	5	0.883	3.712	0.014	0.883	0.750	0.750	See	See
15	KNeighborsClassifier_brute_force	sklearn	predict	100000	1	2	0.009	0.002	0.000	0.009	-1	5	1.000	0.238	0.005	1.000	0.037	0.037	See	See
16	KNeighborsClassifier_brute_force	sklearn	predict	100000	1000	2	2.063	0.004	0.000	0.002	1	100	0.887	3.758	0.020	0.887	0.549	0.549	See	See
17	KNeighborsClassifier_brute_force	sklearn	predict	100000	1	2	0.003	0.000	0.000	0.003	1	100	1.000	0.239	0.004	1.000	0.012	0.012	See	See
18	KNeighborsClassifier_brute_force	sklearn	predict	100000	1000	2	2.797	0.028	0.000	0.003	-1	100	0.887	3.746	0.009	0.887	0.747	0.747	See	See
19	KNeighborsClassifier_brute_force	sklearn	predict	100000	1	2	0.009	0.002	0.000	0.009	-1	100	1.000	0.240	0.005	1.000	0.038	0.038	See	See
20	KNeighborsClassifier_brute_force	sklearn	predict	100000	1000	2	2.046	0.003	0.000	0.002	1	5	0.883	3.711	0.012	0.883	0.551	0.551	See	See
21	KNeighborsClassifier_brute_force	sklearn	predict	100000	1	2	0.003	0.000	0.000	0.003	1	5	1.000	0.241	0.004	1.000	0.012	0.012	See	See
22	KNeighborsClassifier_brute_force	sklearn	predict	100000	1000	2	1.057	0.002	0.000	0.001	1	1	0.845	3.751	0.007	0.845	0.282	0.282	See	See
23	KNeighborsClassifier_brute_force	sklearn	predict	100000	1	2	0.002	0.000	0.000	0.002	1	1	1.000	0.239	0.005	1.000	0.007	0.007	See	See

KNeighborsClassifier_kd_tree

onnx (1.10.1) vs. scikit-learn (1.0.dev0)

Speedup barplots

All estimators share the following parameters: algorithm=kd_tree.

Raw results

predict

	estimator	library	function	n_samples_train	n_samples	n_features	mean_duration_sklearn	std_duration_sklearn	iteration_throughput	latency	n_jobs	n_neighbors	accuracy_score_sklearn	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
0	KNeighborsClassifier_kd_tree	sklearn	predict	1000000	1000	10	1.701	0.009	0.000	0.002	1	5	0.975	121.481	0.000	0.975	0.014	0.014	See	See
1	KNeighborsClassifier_kd_tree	sklearn	predict	1000000	1	10	0.001	0.000	0.000	0.001	1	5	1.000	2.568	0.045	1.000	0.001	0.001	See	See
2	KNeighborsClassifier_kd_tree	sklearn	predict	1000000	1000	10	0.951	0.006	0.000	0.001	-1	5	0.975	121.317	0.000	0.975	0.008	0.008	See	See
3	KNeighborsClassifier_kd_tree	sklearn	predict	1000000	1	10	0.003	0.001	0.000	0.003	-1	5	1.000	2.593	0.023	1.000	0.001	0.001	See	See
4	KNeighborsClassifier_kd_tree	sklearn	predict	1000000	1000	10	0.505	0.012	0.000	0.001	-1	1	0.964	123.475	0.000	0.964	0.004	0.004	See	See
5	KNeighborsClassifier_kd_tree	sklearn	predict	1000000	1	10	0.003	0.001	0.000	0.003	-1	1	1.000	2.567	0.070	1.000	0.001	0.001	See	See
6	KNeighborsClassifier_kd_tree	sklearn	predict	1000000	1000	10	0.903	0.005	0.000	0.001	1	1	0.964	129.775	0.000	0.964	0.007	0.007	See	See
7	KNeighborsClassifier_kd_tree	sklearn	predict	1000000	1	10	0.001	0.000	0.000	0.001	1	1	1.000	2.587	0.044	1.000	0.000	0.000	See	See
8	KNeighborsClassifier_kd_tree	sklearn	predict	1000000	1000	10	5.397	0.102	0.000	0.005	1	100	0.973	134.701	0.000	0.973	0.040	0.040	See	See
9	KNeighborsClassifier_kd_tree	sklearn	predict	1000000	1	10	0.003	0.001	0.000	0.003	1	100	1.000	2.602	0.102	1.000	0.001	0.001	See	See
10	KNeighborsClassifier_kd_tree	sklearn	predict	1000000	1000	10	3.209	0.028	0.000	0.003	-1	100	0.973	127.012	0.000	0.973	0.025	0.025	See	See
11	KNeighborsClassifier_kd_tree	sklearn	predict	1000000	1	10	0.005	0.001	0.000	0.005	-1	100	1.000	2.574	0.042	1.000	0.002	0.002	See	See
12	KNeighborsClassifier_kd_tree	sklearn	predict	1000	1000	2	0.020	0.000	0.001	0.000	1	5	0.923	0.039	0.000	0.923	0.522	0.522	See	See
13	KNeighborsClassifier_kd_tree	sklearn	predict	1000	1	2	0.001	0.000	0.000	0.001	1	5	1.000	0.005	0.000	1.000	0.114	0.114	See	See
14	KNeighborsClassifier_kd_tree	sklearn	predict	1000	1000	2	0.023	0.000	0.001	0.000	-1	5	0.923	0.039	0.001	0.923	0.587	0.587	See	See
15	KNeighborsClassifier_kd_tree	sklearn	predict	1000	1	2	0.002	0.000	0.000	0.002	-1	5	1.000	0.005	0.000	1.000	0.388	0.389	See	See
16	KNeighborsClassifier_kd_tree	sklearn	predict	1000	1000	2	0.021	0.001	0.001	0.000	-1	1	0.895	0.038	0.000	0.895	0.547	0.547	See	See
17	KNeighborsClassifier_kd_tree	sklearn	predict	1000	1	2	0.002	0.000	0.000	0.002	-1	1	1.000	0.005	0.000	1.000	0.402	0.402	See	See
18	KNeighborsClassifier_kd_tree	sklearn	predict	1000	1000	2	0.019	0.000	0.001	0.000	1	1	0.895	0.038	0.000	0.895	0.497	0.497	See	See
19	KNeighborsClassifier_kd_tree	sklearn	predict	1000	1	2	0.001	0.000	0.000	0.001	1	1	1.000	0.005	0.000	1.000	0.115	0.115	See	See
20	KNeighborsClassifier_kd_tree	sklearn	predict	1000	1000	2	0.034	0.000	0.000	0.000	1	100	0.919	0.060	0.000	0.919	0.566	0.566	See	See
21	KNeighborsClassifier_kd_tree	sklearn	predict	1000	1	2	0.001	0.000	0.000	0.001	1	100	1.000	0.005	0.000	1.000	0.117	0.117	See	See
22	KNeighborsClassifier_kd_tree	sklearn	predict	1000	1000	2	0.036	0.005	0.000	0.000	-1	100	0.919	0.060	0.000	0.919	0.597	0.597	See	See
23	KNeighborsClassifier_kd_tree	sklearn	predict	1000	1	2	0.002	0.000	0.000	0.002	-1	100	1.000	0.005	0.000	1.000	0.401	0.401	See	See

HistGradientBoostingClassifier_best

onnx (1.10.1) vs. scikit-learn (1.0.dev0)

Speedup barplots

All estimators share the following parameters: learning_rate=0.01, n_iter_no_change=10.0, max_leaf_nodes=100.0, max_bins=255.0, min_samples_leaf=100.0, max_iter=300.0.

Raw results

predict

	estimator	library	function	n_samples_train	n_samples	n_features	mean_duration_sklearn	std_duration_sklearn	n_iter	iteration_throughput	latency	accuracy_score_sklearn	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
0	HistGradientBoostingClassifier_best	sklearn	predict	100000	1000	100	0.106	0.001	300	0.008	0.000	0.789	0.476	0.013	0.789	0.223	0.223	See	See
1	HistGradientBoostingClassifier_best	sklearn	predict	100000	1	100	0.017	0.001	300	0.000	0.017	1.000	0.400	0.009	1.000	0.041	0.041	See	See